{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KqdfNhcg9lbj"
   },
   "source": [
    "## 1. Install Dependencies and Import Libraries\n",
    "\n",
    "Install Phoenix, LlamaIndex, and OpenAI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install \"arize-phoenix[experimental,llama-index]\" \"openai>=1\" gcsfs accelerate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gyxL_iSP9lbm"
   },
   "source": [
    "Import libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from urllib.request import urlopen\n",
    "\n",
    "import pandas as pd\n",
    "import phoenix as px\n",
    "from gcsfs import GCSFileSystem\n",
    "from llama_index import (\n",
    "    ServiceContext,\n",
    "    StorageContext,\n",
    "    load_index_from_storage,\n",
    "    set_global_handler,\n",
    ")\n",
    "from llama_index.graph_stores.simple import SimpleGraphStore\n",
    "\n",
    "pd.set_option(\"display.max_colwidth\", 1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FpSHFoKm9lbs"
   },
   "source": [
    "## 2. Launch Phoenix\n",
    "\n",
    "You can run Phoenix in the background to collect trace data emitted by any LlamaIndex application that has been instrumented with the `OpenInferenceTraceCallbackHandler`. Phoenix supports LlamaIndex's [one-click observability](https://gpt-index.readthedocs.io/en/latest/end_to_end_tutorials/one_click_observability.html) which will automatically instrument your LlamaIndex application! You can consult our [integration guide](https://docs.arize.com/phoenix/integrations/llamaindex) for a more detailed explanation of how to instrument your LlamaIndex application.\n",
    "\n",
    "Launch Phoenix and follow the instructions in the cell output to open the Phoenix UI (the UI should be empty because we have yet to run the LlamaIndex application)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "session = px.launch_app()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BnuA4nzH9lbw"
   },
   "source": [
    "## 3. Build Your LlamaIndex Application"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kthKy6Zz9lbw"
   },
   "source": [
    "This example uses a `RetrieverQueryEngine` over a pre-built index of the Arize documentation, but you can use whatever LlamaIndex application you like.\n",
    "\n",
    "Download our pre-built index of the Arize docs from cloud storage and instantiate your storage context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_system = GCSFileSystem(project=\"public-assets-275721\")\n",
    "index_path = \"arize-phoenix-assets/datasets/unstructured/llm/llama-index/arize-docs/index/\"\n",
    "storage_context = StorageContext.from_defaults(\n",
    "    fs=file_system,\n",
    "    persist_dir=index_path,\n",
    "    graph_store=SimpleGraphStore(),  # prevents unauthorized request to GCS\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jkqSbNj89lbx"
   },
   "source": [
    "Enable Phoenix tracing within LlamaIndex by setting `arize_phoenix` as the global handler. This will mount Phoenix's [OpenInferenceTraceCallback](https://docs.arize.com/phoenix/integrations/llamaindex) as the global handler. Phoenix uses OpenInference traces - an open-source standard for capturing and storing LLM application traces that enables LLM applications to seamlessly integrate with LLM observability solutions such as Phoenix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "set_global_handler(\"arize_phoenix\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "L5NSlO4g9lby"
   },
   "source": [
    "We are now ready to instantiate our query engine that will perform retrieval-augmented generation (RAG). Query engine is a generic interface in LlamaIndex that allows you to ask question over your data. A query engine takes in a natural language query, and returns a rich response. It is built on top of Retrievers. You can compose multiple query engines to achieve more advanced capability  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ucH1_LeBYwB7"
   },
   "source": [
    "## 4. Import HuggingFaceLLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index import StorageContext\n",
    "from llama_index.llms import HuggingFaceLLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "service_context = ServiceContext.from_defaults(\n",
    "    llm=HuggingFaceLLM(model_name=\"Writer/palmyra-small\", tokenizer_name=\"Writer/palmyra-small\"),\n",
    "    embed_model=\"local\",\n",
    ")\n",
    "\n",
    "index = load_index_from_storage(\n",
    "    storage_context,\n",
    "    service_context=service_context,\n",
    ")\n",
    "query_engine = index.as_query_engine()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-FLHMebj9lby"
   },
   "source": [
    "## 5. Run Your Query Engine and View Your Traces in Phoenix\n",
    "\n",
    "We've compiled a list of commonly asked questions about Arize. Let's download the sample queries and take a look."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "queries_url = \"http://storage.googleapis.com/arize-phoenix-assets/datasets/unstructured/llm/context-retrieval/arize_docs_queries.jsonl\"\n",
    "queries = []\n",
    "with urlopen(queries_url) as response:\n",
    "    for line in response:\n",
    "        line = line.decode(\"utf-8\").strip()\n",
    "        data = json.loads(line)\n",
    "        queries.append(data[\"query\"])\n",
    "queries[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FV06HTNt9lbz"
   },
   "source": [
    "Let's run the first 10 queries and view the traces in Phoenix."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Rt-8CAhE9lb0"
   },
   "source": [
    "And just for fun, ask your own question!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = query_engine.query(\"What is Arize and how can it help me as an AI Engineer?\")\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AFDxtTH09lb0"
   },
   "source": [
    "Check the Phoenix UI as your queries run. Your traces should appear in real time.\n",
    "\n",
    "Open the Phoenix UI with the link below if you haven't already and click through the queries to better understand how the query engine is performing. For each trace you will see a break\n",
    "\n",
    "Phoenix can be used to understand and troubleshoot your by surfacing:\n",
    " - **Application latency** - highlighting slow invocations of LLMs, Retrievers, etc.\n",
    " - **Token Usage** - Displays the breakdown of token usage with LLMs to surface up your most expensive LLM calls\n",
    " - **Runtime Exceptions** - Critical runtime exceptions such as rate-limiting are captured as exception events.\n",
    " - **Retrieved Documents** - view all the documents retrieved during a retriever call and the score and order in which they were returned\n",
    " - **Embeddings** - view the embedding text used for retrieval and the underlying embedding model\n",
    "LLM Parameters - view the parameters used when calling out to an LLM to debug things like temperature and the system prompts\n",
    " - **Prompt Templates** - Figure out what prompt template is used during the prompting step and what variables were used.\n",
    " - **Tool Descriptions** - view the description and function signature of the tools your LLM has been given access to\n",
    " - **LLM Function Calls** - if using OpenAI or other a model with function calls, you can view the function selection and function messages in the input messages to the LLM.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"🚀 Open the Phoenix UI if you haven't already: {session.url}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Cg-W3qaR9lb1"
   },
   "source": [
    "## 6. Export and Evaluate Your Trace Data\n",
    "\n",
    "You can export your trace data as a pandas dataframe for further analysis and evaluation.\n",
    "\n",
    "In this case, we will export our `retriever` spans and evaluate each document retrieval so that we can compute an LLM-assisted precision@k. To learn more about the different span kinds, see the docs on [LLM Traces](https://docs.arize.com/phoenix/concepts/llm-traces)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trace_df = px.Client().get_spans_dataframe('span_kind == \"RETRIEVER\"')\n",
    "trace_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cphedRjK9lb1"
   },
   "source": [
    "Evaluate your retrieval spans and surface problematic spans:\n",
    "\n",
    "- Make LLM calls to classify each retrieved document as relevant or irrelevant to the corresponding query,\n",
    "- Compute the precision@k for k = 1, 2 for each document,\n",
    "- Sort your spans by precision@2 to surface up the most problematic spans.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2AxRis1t9lb2"
   },
   "source": [
    "Phoenix has build-in support for [LLM Evals](https://docs.arize.com/phoenix/concepts/llm-evals). With LLM Evals, you can quickly benchmark the performance of your LLM Application."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
